Indexer fixes and improvements #758

braydonf · 2019-04-12T23:12:32Z

This adds a new module indexer for indexing blocks, and includes several fixes and improvements:

Reduces the size of the txindex from being around 100% of all blocks, down to around 10% of the blocks (Disk I/O efficiency improvements #585).
Fixes DoS by CPU and memory exhaustion for the addrindex and reduces disk usage (CPU exhaustion with address index HTTP API queries #589).
Will correctly distinguish between p2pkh and p2wpkh addresses for the address index, instead of combining the results that use the same public key (Output descriptor not included for address index #738).
Adds an ability to enable the indexes after the initial block download (Interface for indexing plugins #422).
Indexes have been moved into a separate database, and can be disabled or regenerated asynchronously without changing the chain database (Interface for indexing plugins #422 and Memory exhaustion crash with full node startup #682).
Includes an indexing interface for extensibility (Interface for indexing plugins #422).
Resolves issues with duplicate transactions and coins in the mempool (duplicate copies of transactions and UTXOs created from reorganization #499 and Duplicate transactions received when requesting them from bcoin node #594).
Adds clarification on the status of the indexers (missing transactions for address index #734).

The following issues are closed:

Closes: Output descriptor not included for address index #738
Closes: missing transactions for address index #734
Closes: Duplicate transactions received when requesting them from bcoin node #594
Closes: CPU exhaustion with address index HTTP API queries #589
Closes: Disk I/O efficiency improvements #585
Closes: duplicate copies of transactions and UTXOs created from reorganization #499
Closes: Interface for indexing plugins #422

Includes commits from @tuxcanfly, @braydonf and @nodar-chkuaselidze.

tynes · 2019-04-15T22:33:37Z

Could you add migrate/chaindb5to6.js to migrate/latest?

braydonf · 2019-04-15T22:35:02Z

I'm actually thinking we can combine migrate/chain4to5.js and migrate/chain5to6.js, that will do both. In that we can likely update migrate/latest, from what I remember it's not current.

tynes · 2019-04-16T20:28:01Z

It would be really nice to have some indexer information returned from the API. A call to GET / on the node could return an additional field:

"indexer": {
  "addr": {
      "enabled": true,
      "height": 40000
    },
   "tx" : {
      "enabled": false,
      "height" null
    }
}

As @braydonf pointed out, endpoints will return [] if the indexers are not enabled

If an Indexer has some extra methods that enable this data collection (something like indexer.isEnabled, indexer.bestHeight) and all types of Indexers implement those methods, then this object could be dynamically created and updated if additional indexers are added, ie by plugins.

Would be happy to open an issue if this is out of context for this particular PR

braydonf · 2019-04-16T23:07:35Z

@tynes The indexer status has been added, along with the logging of the current status of the indexer, this should help with checking if the indexing is enabled.

lib/indexer/addrindexer.js

lib/indexer/records.js

There was a rare case that a block could be incorrectly added to the indexer if the indexer was disabled during a reorg to a height that matched the height that was expected, and the `sync` method for the indexer wasn't called that would detect the reorg.

Deprecation details are in CHANGELOG.md

braydonf · 2019-05-15T19:50:49Z

Rebased on latest master branch. Further documentation updates are at bcoin-org/bcoin-org.github.io#145.

tuxcanfly

OK.

lib/indexer/indexer.js

CHANGELOG.md

Indexer fixes and improvements

lib/indexer/addrindexer.js

lib/indexer/indexer.js

lib/mempool/addrindexer.js

braydonf force-pushed the indexer branch from 51fe794 to 10ef180 Compare April 12, 2019 23:14

This comment has been minimized.

Sign in to view

braydonf mentioned this pull request Apr 13, 2019

Memory exhaustion crash with full node startup #682

Closed

tuxcanfly mentioned this pull request Apr 13, 2019

refactor indexers #424

Closed

3 tasks

braydonf force-pushed the indexer branch from 62085f5 to 99e5b02 Compare April 15, 2019 22:00

braydonf added indexer stability / efficiency Denial of service, better resource usage labels Apr 15, 2019

braydonf force-pushed the indexer branch 3 times, most recently from 0f23529 to 2fc8439 Compare April 15, 2019 23:03

tuxcanfly mentioned this pull request Apr 16, 2019

Add support for neutrino #370

Closed

3 tasks

braydonf mentioned this pull request Apr 16, 2019

Separate mempool indexer #650

Closed

braydonf force-pushed the indexer branch 4 times, most recently from 031a939 to 4db8d99 Compare April 16, 2019 23:02

braydonf force-pushed the indexer branch 2 times, most recently from c07229b to e61db8c Compare April 17, 2019 17:58

braydonf marked this pull request as ready for review April 17, 2019 18:28

braydonf added the ready for review Ready to be reviewed label Apr 17, 2019

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/addrindexer.js Show resolved Hide resolved

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/records.js Outdated Show resolved Hide resolved

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/records.js Outdated Show resolved Hide resolved

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/records.js Outdated Show resolved Hide resolved

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/records.js Outdated Show resolved Hide resolved

tynes reviewed Apr 17, 2019

View reviewed changes

lib/indexer/records.js Outdated Show resolved Hide resolved

braydonf and others added 7 commits May 15, 2019 12:11

migrate: update latest migration

5111af0

migrate: upgrade from chaindb 4 to 6

677b45f

changelog: update with latest migration

a6fd1be

node: simplify index-prefix option

ce16bc9

changelog: update with new indexer fields for info endpoint

372b9f9

blockchain: cleanup deprecated methods

662a68c

Deprecation details are in CHANGELOG.md

braydonf force-pushed the indexer branch from 990c3db to 662a68c Compare May 15, 2019 19:39

braydonf mentioned this pull request May 16, 2019

node-http: use addresses instead of strings #773

Closed

tuxcanfly approved these changes May 16, 2019

View reviewed changes

lib/indexer/indexer.js Outdated Show resolved Hide resolved

lib/indexer/indexer.js Outdated Show resolved Hide resolved

tuxcanfly reviewed May 16, 2019

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

braydonf added 3 commits May 16, 2019 10:25

indexer: update and fix documentation

34e8f22

node: pass options from getTXByAddress to getMetaByAddress

7e9d00a

changelog: update and fix chain changes

6c497d4

braydonf merged commit 6c497d4 into bcoin-org:master May 16, 2019

braydonf added a commit to braydonf/bcoin that referenced this pull request May 16, 2019

Merge pull request bcoin-org#758 from braydonf/indexer

7e49be2

Indexer fixes and improvements

braydonf deleted the indexer branch May 16, 2019 18:36

braydonf removed the ready for review Ready to be reviewed label May 16, 2019